Development and Pilot Evaluation of a Standardized and Accessible Rubric (OMNI) for Assessing Generative AI in Health Care Contexts: Four-Stage Mixed Methods Study

Introduction

Generative AI is rapidly expanding across health applications, from clinical decision support and medical reasoning to patient-facing education, clinical documentation, ambient monitoring, continuing education, and the automated generation of health-related images and video [1-3]. Public access to large language models (LLMs), through tools such as ChatGPT (OpenAI), Gemini (Google), and integrated search engines, is also fundamentally shifting how consumers seek and receive health information [4]. The scale of this shift is substantial. OpenAI reports that health is among the most common uses of ChatGPT, with millions of users asking health and wellness questions each week [5], while Microsoft receives more than 50 million health-related sessions daily through Bing and Copilot [6]. Overall, the pace of development is outstripping both regulatory frameworks and the health sector’s capacity to systematically evaluate the quality and safety of accessible tools, both publicly and within health care [7].

Evaluation of generative AI in health contexts remains largely unstandardized. A systematic review of 519 studies, many of which focused on medical knowledge and communication tasks, found wide variation in evaluation methods, limiting the ability to compare findings or aggregate evidence into actionable guidance [8]. The consequences of failing to translate evidence into action are not abstract. As it stands, LLMs may exaggerate research findings [9] and can be repurposed as health disinformation tools [10]; consumer-facing AI apps are assuming quasi-professional health roles [11]; and an independent evaluation of OpenAI’s ChatGPT Health has identified a lack of clarity on its purpose and substantial limitations in its capabilities [12]. Identifying and characterizing such failures requires structured expert assessment not only of accuracy but also across the dimensions that determine whether AI-generated health content is fit for purpose, including safety, contextual appropriateness, empathetic communication, uncertainty recognition, and freedom from coercive framing [8,13-15]. Yet, existing evaluation frameworks remain inconsistent and do not provide a practical, expert-accessible instrument that integrates these dimensions in a standardized way across diverse health care use cases. Without such a framework, the growing body of evidence on generative AI in health remains difficult to synthesize, compare, and translate into trustworthy implementation guidance [8,14]. Furthermore, while emerging approaches propose using AI itself to evaluate AI-generated content at scale, these methods still depend on expert-derived criteria and human-validated benchmarks to anchor their judgments [13,16,17].

Human evaluation remains an important component of generative AI evaluation in health care, particularly for dimensions requiring clinical judgment, such as safety, contextual appropriateness, and communication quality [8,14,18]. However, across the published literature, human evaluations remain heterogeneous, largely unstandardized, and unscaled, while established health information quality instruments that predate generative AI were not designed for dynamic, context-dependent outputs or dimensions particularly salient to these systems, including uncertainty recognition, instruction following, contextual tailoring, and potentially coercive behavior [8,14,18-21]. A practical, expert-accessible instrument that standardizes what is assessed and how it is scored across diverse health care use cases is therefore needed to support consistent and comparable evaluation and provide a credible foundation for quality assurance of generative AI in health. This study aimed to develop and pilot a structured rubric for expert-led evaluation of generative AI outputs across diverse health care contexts, spanning text, image, and video modalities and encompassing clinical decision support, clinical note generation, patient and clinician communication, education, and administration. The result is OMNI, a standardized and accessible rubric comprising defined evaluation domains, structured assessment questions, domain-specific scoring indicators, and supporting guidance for expert-led evaluation of generative AI outputs in health care contexts.

Methods

Study Design

This study used a 4-stage, mixed methods design to develop and pilot a structured rubric for human-led evaluation of generative AI outputs in health contexts (Figure 1). Stage 1 involved a structured literature synthesis to inform the development of an initial draft rubric. Stage 2 involved subject-matter experts to refine the draft rubric and establish consensus on evaluation domains, definitions, scoring criteria, and usability. Stage 3 involved pilot application of the refined rubric in 4 separate health care scenarios to assess feasibility and applicability. Stage 4 captured assessor feedback to inform final rubric refinements and final rubric design. In this manuscript, “draft rubric” refers to the initial design developed in stage 1, “pilot rubric” refers to the refined version applied in stage 3, and “OMNI” refers to the final version following pilot application and assessor feedback. OMNI is available as Note S1 in Multimedia Appendix 1 and on GitHub [22].

‎

Stage 1: Literature Synthesis and Draft Rubric Development

A structured literature synthesis was conducted to identify existing approaches to human-led evaluation of generative AI in health care and to inform the development of an initial draft rubric.

Data Collection

Searches were performed in PubMed and Google Scholar for English-language studies published between January 1, 2022, and July 31, 2025, using terms related to generative AI, LLMs, evaluation, assessment, benchmarking, and health care. The full search strategy is provided in Note S2 in Multimedia Appendix 1, with reporting guided by PRISMA-S (Preferred Reporting Items for Systematic Reviews and Meta-Analyses literature search extension) and the completed checklist provided in Checklist 1. Studies were included if they involved human evaluation of generative AI outputs in health-related contexts and either (1) synthesized evaluation domains or frameworks (systematic, meta-analytic, or narrative reviews) or (2) described the development and application of human-led benchmarking or structured evaluation tools. Given the volume of narrowly scoped evaluations, priority was given to studies proposing generalizable or cross-context evaluation frameworks. Reference lists and citations of identified studies were screened for additional relevant publications, with supplementary searches conducted using generative AI–assisted research tools (Gemini Deep Research [Google], Perplexity [Perplexity AI, Inc], and Elicit [Elicit Research, PBC]).

Data Analysis

The literature synthesis was conducted collaboratively by 3 authors (EC, BDM, and AMH), who jointly reviewed eligible studies and extracted recurring evaluation domains, scoring approaches, and practical considerations for clinician-led assessment. Foundational sources proposing comprehensive or influential evaluation approaches were used to develop an initial draft rubric comprising candidate domains, domain definitions, and proposed scoring approaches. This draft rubric was carried forward into stage 2, with the literature synthesis continuing to inform interpretation and refinement throughout the process.

Stage 2: Expert Refinement

Stage 2 aimed to refine the draft rubric developed in stage 1 through structured expert input and to establish consensus on evaluation domains, domain definitions, scoring criteria, and practical application.

Data Collection

Three rounds of review were conducted to reach consensus among a panel of 23 subject-matter experts, all of whom were study investigators engaged in the rubric development process, including physicians (generalists and specialists in hematology, oncology, and neurology), pharmacists, psychologists, health care administrators, and researchers with expertise in AI safety, health informatics, and the evaluation of AI in health care.

Over 3 rounds, rubric materials were circulated electronically via email and iteratively revised based on panel feedback, with panelists providing input via structured email responses and tracked changes to draft documents. Each round used guided questions to elicit feedback on domain relevance and completeness, the clarity of domain definitions, the appropriateness of scoring criteria, usability, and anticipated applications in health care contexts. Example generative AI outputs for health-related scenarios were included in the emails to ensure that the feedback received was grounded in practical use (see Note S3 in Multimedia Appendix 1 for both guided questions and examples for each round).

In round 3, the rubric was circulated via both document-based feedback and an online survey (Qualtrics; Qualtrics, LLC) to assess readiness for pilot testing and to identify any substantive objections.

Data Analysis

For this developmental stage, agreement to proceed to pilot testing was a priori defined by the team as the absence of substantive objections to rubric structure, content, or readiness for pilot application. As no substantive objections were raised in round 3, the refined rubric proceeded to application in stage 3.

Stage 3: Pilot Application in 4 Separate Health Care Scenarios

Stage 3 Overview

Stage 3 had 2 objectives: first, to evaluate the rubric’s applicability within different AI use cases, including its ability to support within-scenario AI model comparisons, and second, to obtain assessor insights to inform the final rubric design. The rubric was applied to generative AI in four separate health care scenarios: (1) responses to health-related questions posted on Reddit, (2) responses to prompts requesting generation of health-related image and video content, (3) responses to health care professional continuing education questions, and (4) responses to prompts requesting automated preparation of physician-to-physician clinical documentation.

Data Collection

Overview

For each scenario, each generative AI output was created within a unique chat session, with all chat history and memory settings disabled to ensure consistency; a detailed overview of output generation is provided in Table S1 in Multimedia Appendix 1. All generative AI input-output pairs were collected in full, with formatting and hyperlinks preserved, and presented to assessors for evaluation within the Qualtrics platform. For each scenario, study investigators with relevant health professional expertise were assigned as assessors based on their specific clinical experience and independently evaluated the AI outputs. Fifteen study investigators acted as assessors in stage 3, of whom 2 were also members of the stage 2 expert panel. Where necessary, a brief assessor calibration meeting was held to familiarize assessors with the rubric structure and scenario-specific question types prior to independent assessment. Assessors were blinded to the generative AI model responsible for each input-output pair, acknowledging that some AI outputs may explicitly reference the model’s name.

Health Care Scenario 1: Responding to Health-Related Questions Posted on Reddit

On June 3, 2025, the 50 most recently posted health care–related questions were extracted from Reddit’s r/AskDocs forum, together with the most upvoted response tagged as “Physician.” According to r/AskDocs, users with this tag have submitted professional credentials to Reddit moderators; these credentials were not independently verified by the study team, and specialty and clinical experience were not available [23]. The primary purpose of this scenario was to evaluate the rubric using real-world health questions. Physician-tagged replies were included as examples of publicly available responses to these questions, rather than as benchmarks of physician performance or formal clinical care. On August 18, 2025, each question was submitted to GPT-5 Auto (via ChatGPT) [24] and Gemini-2.5-Pro (via Gemini web app) [25]. The resulting 100 generative AI input-output pairs, along with the 50 “Physician”-tagged responses, were independently evaluated by 4 medical practitioner researchers (MA, HC, J Goonan, and PM) using the pilot rubric.

Health Care Scenario 2: Creation of Health-Related Image and Video Content

Twenty synthetic prompts requesting generation of health-related images and videos were developed by 3 authors (EC, BDM, and AMH). The prompt set comprised 10 image-based and 10 video-based requests, spanning educational and informational content on common medical conditions and scenarios, as well as adversarial requests that challenged established health guidelines or called for potentially unsafe health actions. The complete list of prompts is provided in Note S4 in Multimedia Appendix 1.

On November 21, 2025, image-based prompts were submitted to GPT-5.1 Auto (via ChatGPT) [26] and Gemini-3-Pro Image (via Gemini web app) [27], while video-based prompts were submitted to Sora 2 (via Sora web app; OpenAI) [28] and Veo 3.1 (via Gemini web app) [29], reflecting modality-specific deployment of OpenAI and Google generative models, respectively. The resulting generative AI input-output pairs were independently evaluated by 4 clinical pharmacist researchers (KD, SF, J Gandhi, and NLS) using the pilot rubric. For the primary within-scenario analysis, image and video outputs were pooled to enable a broader multimodal comparison within the scope of the pilot, while modality-specific comparisons were conducted as exploratory analyses.

Health Care Scenario 3: Continuing Education Questions for Health Care Professionals

Twenty pharmacy practice questions relevant to continuing education were sourced from Public Health in Pharmacy Practice: A Casebook (2nd edition) [30]. On August 27, 2025, each question was submitted to GPT-5 Auto (via ChatGPT) [24] and Gemini-2.5-Pro (via Gemini web app) [25]. The resulting generative AI outputs were independently evaluated by 3 pharmacy practice academics (EC, SF, and J Gandhi) using the pilot rubric. The full set of questions is provided in Note S5 in Multimedia Appendix 1.

Health Care Scenario 4: Automated Generation of Physician-to-Physician Clinical Documentation

Synthetic oncology case notes and the associated prompts for generating physician-to-physician letters were adopted from Shahnam et al [31]. The case notes reflect information recorded during an initial oncology consultation, and the prompts instruct generative AI models to generate a structured letter from an oncologist to a primary care physician based on these case notes. On August 28, 2025, 10 synthetic oncology case notes, together with the physician-to-physician letter generation prompt, were submitted to GPT-5 Auto (via ChatGPT) [24] and Claude Opus 4.1 (via Claude.ai; Anthropic) [32]. The resulting generative AI input-output pairs were independently evaluated by 4 medical oncologist researchers (NH, MH, AS, and JT) using the pilot rubric. The full set of case notes and prompts used in this scenario is provided in Note S6 in Multimedia Appendix 1.

Data Analysis

To characterize assessor agreement within each scenario, quantitative rubric data were evaluated using mean pairwise percentage agreement and Gwet’s AC2 with quadratic weights, selected for its lower sensitivity to prevalence imbalance while retaining a familiar scale for assessing agreement [33]. Within each scenario, exploratory comparisons of model performance were conducted on the pooled assessor responses using chi-square tests of the broad agreement categories, excluding responses coded as not applicable. Benjamini-Hochberg–adjusted P values were used to determine statistical significance at P<.05, with both unadjusted and adjusted P values and Cramér V estimates with 95% CIs reported in Multimedia Appendix 1. For descriptive analyses, the 4-point Likert scale was collapsed into broad agreement (agree and strongly agree) and broad disagreement (disagree and strongly disagree) categories. As this was a pilot feasibility study, no formal a priori power calculation was performed; instead, scenario-specific numbers of input-output pairs were determined according to the volume and complexity of material that could feasibly be evaluated, while assessor numbers reflected the availability of study investigators with relevant clinical expertise. Accordingly, model comparisons were considered hypothesis generating. Qualitative assessor comments were analyzed thematically using NVivo (version 14.24.3; Lumivero LLC). EC undertook the primary coding, and DK, who had not participated in rubric development, expert refinement, or pilot assessment, subsequently reviewed the coding framework and its application to the comments while blinded to assessor identity. Comments were recorded within rubric domains at the point of assessment, and an inductive approach was used to analyze free-text responses within each domain, identifying recurring patterns and developing themes directly from the data.

Stage 4: Assessor Feedback and Final Rubric Design

Following completion of the pilot evaluations in the 4 separate health care scenarios, the study investigators who served as stage 3 assessors completed a structured feedback questionnaire to inform final rubric refinement.

Data Collection

The questionnaire examined how well the rubric functioned in each scenario, whether any domains could be excluded to reduce the scoring burden, and the clarity of the scenario-appropriate reference outputs. Assessors were also asked about the usefulness of the onboarding and calibration meeting, the need for worked exemplars, and whether any domain definitions or examples required revision or expansion.

Additional items explored the perceived value of optional future “plug-in” domains for project-specific evaluations, whether assessors would reuse or recommend the rubric, and any changes that could improve its suitability. The full questionnaire is provided in Note S7 in Multimedia Appendix 1.

Data Analysis

Responses included categorical selections with optional free-text explanations. Feedback was synthesized by the study team and used to inform final revisions to the rubric.

Ethical Considerations

This study was approved by the Flinders University Human Research Ethics Committee (Project ID: 7800), in accordance with the National Statement on Ethical Conduct in Human Research. Individuals involved in the expert panel, pilot application, and assessor feedback stages participated as study investigators; accordingly, participant consent was not required. No compensation was provided to study investigators for their participation.

Results

Stage 1: Literature Synthesis and Draft Rubric Development

The literature synthesis identified 69 publications, including systematic and scoping reviews, evaluation frameworks, and primary benchmarking studies, on human-led evaluation of generative AI outputs in health care, as well as related literature informing benchmarking tool development (Table S2 in Multimedia Appendix 1 [8,10,11,13,14,21,34-96]).

Three sources were particularly influential in shaping the initial draft rubric: OpenAI’s HealthBench framework [13], a systematic review by Bedi et al [8] of 519 LLM evaluation studies in health care, and the QUEST framework for human evaluation of LLMs in health care [14]. Taken together, these sources highlighted heterogeneity in existing evaluation methods and identified recurring themes relevant to clinician-led assessment, including factual accuracy, completeness, communication, contextualization, adherence to instructions, and safety-related behaviors such as appropriate referencing, recognition of uncertainty, and avoidance of coerciveness and sycophancy.

The stage 1 draft rubric adopted an axis-based structure aligned with HealthBench and used a continuous visual analog scoring approach from −10 (strongly disagree) to +10 (strongly agree). The draft included 6 evaluation axes: accuracy, completeness and safety, communication quality, context awareness, instruction following, and referencing. Each axis was accompanied by preliminary positive and negative scoring anchors to guide interpretation. In this initial version, the axes were intentionally broad, with each axis encompassing multiple related behavioral constructs as per HealthBench. This approach allowed for fewer evaluation categories, but individual axes implicitly assessed several dimensions simultaneously. The stage 1 draft rubric circulated to the expert panel is provided in Note S8 in Multimedia Appendix 1.

Stage 2: Expert Refinement

Across 3 rounds, the stage 1 draft rubric was iteratively refined by the expert investigator panel. The rubric evolved from broad axes scored on a visual analog scale to discrete evaluation domains assessed using a 4-point Likert scale (strongly disagree to strongly agree). Additional domains, including Safety, Empathetic, Recognizes Uncertainty, and Avoids Coerciveness, were introduced to separate key constructs that had previously been embedded within broader HealthBench axes. Further changes arising from panel review included a “not applicable” (NA) response option, removal of the neutral response option to require a directional judgment where the domain was applicable, enhanced domain-level scoring indicators, and detailed assessor guidance and instructional materials. At the end of the third round, no expert raised objections to advancing the rubric to pilot application. The draft rubrics from each expert review round and the pilot rubric, including verbatim assessment questions, detailed scoring indicators, and instructional and training materials, are provided in Note S9 in Multimedia Appendix 1. The resulting pilot-ready rubric comprised 10 evaluation domains (Table 1), each assessed with a standardized, domain-specific assessment question and structured scoring guidance on a 4-point Likert scale (strongly disagree, disagree, agree, and strongly agree). As specified in the rubric instructions, assessors were instructed to first conceptualize an expert reference output appropriate to the scenario and intended audience before applying domain-level ratings. Domain scoring followed a standardized workflow in which assessors began at strongly disagree and moved upward only if higher-category criteria were not violated; the final rating reflected the most serious limitation observed within that domain. Assessors also documented brief qualitative notes under Key Considerations to justify ratings, including any use of “NA.”

Stage 3: Pilot Application in 4 Separate Health Care Scenarios

Overview

In 4 separate health care scenarios, 250 generative AI and human-authored input-output pairs were evaluated by 15 assessors, yielding 9600 domain-level ratings. Scenario-level response distributions by domain are summarized in Figures 2-5. Gwet’s AC2, calculated using quadratic weights to account for the ordinal distance between Likert ratings and excluding NA, ranged from 0.49 to 0.80 across the 4 scenarios (Table S3 in Multimedia Appendix 1). The mean pairwise agreement across 5 response categories (4-point Likert plus NA) ranged from 30.8% to 53.4% across the 4 scenarios, increasing to 59.3% to 79.6% when Likert responses were collapsed into broad disagreement and agreement categories (with NA retained; Table S4 in Multimedia Appendix 1). This increase indicated that a substantial proportion of assessor variation reflected differences in rating strength within the same overall direction, rather than opposing assessments.

‎

‎

‎

‎

Health Care Scenario 1: Responding to Health-Related Questions Posted on Reddit

In scenario 1, Gemini-2.5-Pro (via Gemini web app) achieved >90% broad agreement in 9 of 10 evaluated domains (with Referencing as the exception), while GPT-5 Auto (via ChatGPT) achieved this in 7 of 10 domains (with Recognizes Uncertainty, Empathetic, and Referencing as exceptions). Comparisons of broad agreement proportions between Gemini-2.5-Pro (via Gemini web app) and GPT-5 Auto (via ChatGPT) identified significant differences for Referencing (χ²₁=46.6; P<.001; Cramér V=0.44, 95% CI 0.32‐0.57) and Empathetic (χ²₁=15.0; P<.001; Cramér V=0.19, 95% CI 0.10‐0.29; Table S5 in Multimedia Appendix 1), with GPT-5 Auto (via ChatGPT) receiving a higher proportion of disagreement ratings in both domains (Figure 2). Physician-tagged Reddit responses received ≥45% broad disagreement across 9 of 10 evaluated domains. Compared with the AI-generated responses, the physician-tagged Reddit replies had a significantly higher proportion of negative ratings in all domains except for Referencing, where the performance was similar (Table S5 in Multimedia Appendix 1). “NA” was selected for >30% of Referencing ratings in scenario 1. Detailed plots of response distributions by each assessor are shown in Figures S1 to S3 in Multimedia Appendix 1.

Health Care Scenario 2: Creation of Health-Related Image and Video Content

In scenario 2, the Google model pair comprised Gemini-3-Pro Image (via Gemini web app) for images and Veo 3.1 (via Gemini web app) for video, while the OpenAI model pair comprised GPT-5.1 Auto (via ChatGPT) for images and Sora 2 (via Sora web app) for video. Both model pairs received >30% broad disagreement in 9 of 10 evaluated domains, with Avoids Coerciveness as the exception (Figure 3). Comparisons of the Google and OpenAI model pairs identified a significant difference for Referencing (χ²₁=9.9; P=.02; Cramér V=0.29, 95% CI 0.11‐0.47; Table S6 in Multimedia Appendix 1), with the OpenAI model pair receiving a higher proportion of negative ratings for this domain. “NA” was selected for >30% of ratings for Recognizes Uncertainty, Empathetic, and Referencing in scenario 2. Detailed plots of response distributions by each assessor are shown in Figures S4 and S5 in Multimedia Appendix 1, and examples of image and video outputs are shown in Figure S6 in Multimedia Appendix 1. Exploratory comparisons stratified by image- and video-generation modalities are reported in Table S6 in Multimedia Appendix 1.

Health Care Scenario 3: Continuing Education Questions for Health Care Professionals

In scenario 3, Gemini-2.5-Pro (via Gemini web app) achieved >90% broad agreement in 7 of 10 domains (with Recognizes Uncertainty, Completeness, and Referencing as exceptions), whereas GPT-5 Auto (via ChatGPT) achieved this threshold in only 3 domains (Empathetic, Context Aware, and Avoids Coerciveness; Figure 4). Comparisons between Gemini-2.5-Pro (via Gemini web app) and GPT-5 Auto (via ChatGPT) identified significant differences for Referencing (χ²₁=33.2; P<.001; Cramér V=0.54, 95% CI 0.36‐0.73), Completeness (χ²₁=7.4; P=.03; Cramér V=0.25, 95% CI 0.07‐0.43), and Safety (χ²₁=6.1; P=.04; Cramér V=0.24, 95% CI 0.05‐0.42; Table S7 in Multimedia Appendix 1), with GPT-5 Auto (via ChatGPT) receiving a higher proportion of negative ratings in all 3 domains. Detailed plots of response distributions by each assessor are shown in Figures S7 and S8 in Multimedia Appendix 1.

Health Care Scenario 4: Automated Generation of Physician-to-Physician Clinical Documentation

In scenario 4, Claude Opus 4.1 (via Claude.ai) achieved >90% broad agreement in 6 of 10 domains (with Accuracy, Communicates Clearly, Empathetic, and Referencing as exceptions), while GPT-5 Auto (via ChatGPT) achieved this in 5 of 10 domains (with Accuracy, Recognizes Uncertainty, Completeness, Empathetic, and Referencing as exceptions; Figure 5). Comparisons between Claude Opus 4.1 (via Claude.ai) and GPT-5 Auto (via ChatGPT) did not identify statistically significant differences in any domain (Table S8 in Multimedia Appendix 1). “NA” was selected for >50% of ratings for Referencing and Avoids Coerciveness in scenario 4. Detailed plots of response distributions by each assessor are shown in Figures S9 and S10 in Multimedia Appendix 1.

Qualitative Assessor Responses Across the 4 Health Care Scenarios

Full thematic categorization of the 4010 free-text “Key Considerations” comments is provided in Figures S11 to S16 in Multimedia Appendix 1, with the 6 most frequently coded themes per scenario summarized in Figure 6. Across scenarios, absence of references emerged as a recurrent top 6 theme in 3 of 4 scenarios (scenario 1: n=33; scenario 2: n=85; and scenario 3: n=44). Factual errors or incorrect statements were also commonly identified (scenario 1: n=40; scenario 2: n=58). In scenario 1, additional frequent negative themes included irrelevant or overly verbose information (n=35), failure to seek clarification (n=22), and failure to acknowledge user emotions (n=19), alongside positive comments noting factual or clinically sound content (n=19). In scenario 2, the remaining top themes included negative comments on failure to identify missing contextual information or ask clarifying questions (n=54), inadequate safety or risk mitigation (n=46), accuracy undermined by omission of required information (n=45), and visual or image inaccuracies with potential for confusion or harm (n=43). In scenario 3, omission of key expected components (n=73) was the dominant additional negative concern, while clear and effective communication (n=53), adequate safety responses (n=36), adherence to specific prompt requirements (n=36), and factual or clinical soundness (n=35) were frequently identified as positives. In scenario 4, additional prominent negative themes included insensitivity or lack of patient perspective (n=24) and structural or logical inconsistencies (n=22), while positive themes included clear and effective communication (n=29), factual or clinical soundness (n=24), respectful tone and language (n=24), and acknowledgment of relevant clinical uncertainty (n=23).

‎

Stage 4: Assessor Feedback and Final Rubric Design

All stage 3 assessors (n=15), including 4 medical practitioners, 7 clinical pharmacists, and 4 medical oncologists, completed the post-pilot feedback questionnaire. Key findings are briefly summarized below, with a detailed summary reported in Table S9 in Multimedia Appendix 1. Of the assessors, 20% (n=3) reported themselves as “very confident” in forming scenario-appropriate reference outputs prior to scoring, with 27% (n=4) “confident,” and 53% (n=8) “somewhat confident”; no assessor reported “not confident.” Regarding the helpfulness of an onboarding meeting with worked exemplars, 87% (n=13) indicated they were helpful, with comments highlighting the importance of guiding assessor scoring across domains. All assessors (n=15, 100%) were in consensus that the domain definitions and examples were appropriate. In total, 60% (n=9) of assessors indicated that, for certain projects, certain domains within the rubric may be warranted to be excluded from evaluation, while 80% (n=12) indicated that for certain projects, there may be value in creating additional “plug-in” domains beyond the 10 currently incorporated (eg, hallucination resistance, reproducibility, latency, or cost). Overall, 67% (n=10) of assessors reported it “worked well” for their scenario; 33% (n=5) reported it “worked somewhat,” with no assessor indicating it did not work; 53% (n=8) of assessors reported they would reuse or recommend the rubric without any changes; and 47% (n=7) reported they would reuse or recommend it with minor changes. No assessor reported they would not reuse or recommend the rubric.

On the basis of the assessor feedback, the rubric underwent minor refinements to improve the clarity of the rubric instructions, specifically regarding guidance on scenario-level domain selection, the importance of a pre-evaluation onboarding meeting, and minor refinements to domain-level scoring indicators. The final rubric, hereafter referred to as OMNI, is provided in Note S1 in Multimedia Appendix 1.

Discussion

This study presents OMNI, a structured, expert-accessible rubric for evaluating generative AI outputs in diverse health contexts, spanning 10 domains—Accuracy, Safety, Recognizes Uncertainty, Completeness, Communicates Clearly, Empathetic, Context Aware, Referencing, Follows Instructions, and Avoids Coerciveness—each assessed on a 4-point Likert scale and supported by domain-level scoring indicators. OMNI was developed through a literature synthesis, consensus with 23 expert panelists, and refinement following pilot application and structured assessor feedback. Fifteen assessors applied OMNI in 4 separate health care scenarios selected to reflect current and emerging applications of generative AI: responding to public health questions on social media, generation of health-related images and videos, continuing education for health care professionals, and automated physician-to-physician clinical documentation. Different generative AI platforms were evaluated in each scenario, and across 9600 domain-level ratings and more than 4000 free-text justifications, OMNI identified substantial and consistent limitations in AI performance. In over half of all domain-level evaluations (42/80, 52.5%), at least 10% of assessors rated AI input-output quality negatively (disagree or strongly disagree). Weaknesses were most pervasive in Referencing (negative ratings in all 8 model evaluations), Recognizes Uncertainty (6/8), Empathetic, Completeness, and Accuracy (each 5/8). Some of these observed weaknesses were not unexpected; for example, poor referencing performance is consistent with recent evidence that many LLM-generated medical responses are only partially supported, and sometimes contradicted, by the sources they cite [84]. Of the assessed scenarios, health-related image and video outputs had the greatest number of domains with substantial negative ratings, with both evaluated models exceeding 30% negative ratings in 9 of 10 domains. Overall, OMNI enabled systematic comparison of AI performance within individual scenarios, differentiated model strengths and weaknesses across the evaluation domains, and generated structured, expert-authored feedback that goes beyond what aggregate scoring alone can capture. All 15 assessors, who had received onboarding and applied OMNI within a supported pilot study, confirmed they would reuse or recommend OMNI for future evaluations, with a subset identifying minor refinements that were incorporated into the final design. Taken together, these findings suggest that OMNI provides a useful starting point for structured expert evaluation of generative AI outputs across health contexts.

The current evaluation landscape for generative AI in health contexts encompasses both automated and human-led approaches. Automated evaluation methods, including standard performance metrics, benchmark datasets, and LLM-as-a-Judge frameworks, offer scalability and reproducibility but often rely on synthetic tasks, static datasets, or idealized scenarios or examinations that may not reflect real-world clinical interactions or how users interpret and act on AI-generated outputs [8,14,85,88,97,98]. In contrast, human-led evaluation is increasingly regarded as essential for assessing dimensions such as clinical safety, contextual appropriateness, and communication quality [14]. However, existing studies demonstrate substantial variability in how such evaluations are designed and reported. A recent systematic review of 519 health care LLM evaluation studies highlighted wide variation in evaluation domains, scoring approaches, and reporting methods, limiting the ability to compare findings across studies or assess cumulative progress in model performance [8]. In addition, engaging clinicians with the expertise and capacity to perform structured evaluations remains challenging, further contributing to inconsistency in current assessment practices [14]. Together, these limitations underscore the need for a standardized, expert-accessible framework to support consistent evaluation of generative AI outputs across health contexts. Despite this recognized need, no single evaluation tool is currently applicable across the breadth of generative AI applications in health contexts [8,13,14,86,88-90]. OMNI was developed to help minimize this gap. Its development was informed by a literature synthesis drawing on several foundational frameworks, including OpenAI’s HealthBench framework, which demonstrated the value of physician-authored rubrics across behavioral evaluation axes [13], the systematic review by Bedi et al [8] highlighting the need for standardized multidomain evaluation approaches, and the QUEST framework proposed by Tam et al [14] for structured human evaluation of LLMs in health care.

Building on these foundations, OMNI translates key evaluation constructs into a practical tool, refined through expert consensus, designed to support structured, accessible, expert-led evaluation of generative AI outputs across diverse health contexts. In doing so, OMNI differs from established information quality instruments such as DISCERN [19], which was developed for written consumer health information and not designed to evaluate AI-specific content [20]; from AI-specific quality scales such as the Quality Analysis of Medical Artificial Intelligence, which was developed and applied within narrow clinical contexts [21]; from broader frameworks such as QUEST, which define dimensions and processes for evaluation but require application-specific operationalization of evaluation guidelines and scoring criteria [14]; and from HumanELY, which seeks to standardize definitions and improve accessibility but is presented without empirical testing in health care contexts [18]. OMNI extends these approaches by combining standardized evaluation domains and scoring guidance with empirical application in 4 separate health care scenarios spanning text, image, and video outputs, providing initial evidence of its applicability across distinct generative AI use cases. Importantly, its application in this study demonstrated that a common, accessible standard could support structured expert-led evaluation and meaningful within-scenario comparison across diverse health care applications, while potentially reducing reliance on bespoke assessment approaches and providing a foundation for more scalable human-led and hybrid evaluation in health care.

The 10 domains incorporated within OMNI were derived from a literature synthesis and refined through expert consensus, ensuring that the framework represents key behavioral dimensions of generative AI while reflecting both established evaluation constructs and clinician priorities. Early rubric iterations used broader evaluation axes and continuous scoring scales; however, expert feedback indicated that separating constructs into discrete domains with a structured Likert scoring approach improved clarity and usability for assessors. This separation does not imply that the domains are wholly independent, and some overlap is expected because dimensions of health care quality are often interrelated rather than mutually exclusive. In addition to domain-level scoring, OMNI incorporates a mixed methods assessment structure combining Likert scale ratings with qualitative “Key Considerations” comments. This design enables assessors to rate output quality and articulate the reasoning behind their judgments. Previous methodological research has shown that open-ended feedback alongside structured survey responses can improve the interpretation of quantitative findings and support refinement of evaluation frameworks [99,100]. In this study, the Key Considerations comments provided clinicians with an opportunity to document specific examples of both strengths and limitations in AI outputs (such as misalignment with local clinical guidelines, or missed opportunities to educate users about risk), thereby generating granular, clinician-generated insights that would otherwise remain obscured within aggregate Likert scores. Collectively, these design choices were intended to support structured and interpretable evaluation of generative AI outputs while maintaining practical usability for clinician assessors. Importantly, OMNI is accompanied by detailed instructional materials that were iteratively refined across the expert consensus rounds and in response to pilot assessor feedback. These materials guide critical procedural steps, including how assessors formulate expert reference outputs, the standardized scoring workflow, a priori domain selection to ensure relevance and reduce assessor burden, and pre-evaluation calibration meetings to align the assessment team on domain interpretation and scoring approach. Pilot assessors consistently endorsed the value of these components, with 87% indicating the onboarding meeting with worked exemplars was helpful in guiding their scoring. While the rubric structure itself is intentionally simple, the accompanying guidance is integral to its consistent and rigorous application across research teams and evaluation contexts.

This study has several methodological strengths. OMNI was developed through a structured, sequential multistage methodology that incorporated literature synthesis, expert consensus, pilot application across multiple health care scenarios, and structured assessor feedback. This design enabled each stage to inform the next. The literature synthesis established the initial rubric framework, which was refined through expert input into the pilot rubric; pilot application examined its use across diverse health care scenarios, while assessor ratings, qualitative Key Considerations, and post-pilot feedback informed interpretation of its applicability, limitations, and areas for refinement. Additionally, engagement with clinicians and interdisciplinary experts during rubric development and testing ensured that the framework reflects real-world evaluation priorities and practical usability considerations. Stage 3 assessors represented a diverse range of clinical roles, jurisdictions, cultural backgrounds, and years of clinical experience. Their use of OMNI, together with positive post-pilot feedback, supports its applicability across diverse assessor groups. Application of OMNI across diverse scenarios (including public health communication, multimodal content generation, professional education, and clinical documentation) demonstrated its adaptability across current and emerging generative AI use cases [3,90,101]. OMNI was also applied across multiple generative modalities, including text, image, and video outputs, reflecting the increasing relevance of multimodal AI systems in health care [102,103].

It is noted that interrater agreement varied across domains, reflecting the inherent subjectivity of expert-led evaluation and differences in clinical interpretation among assessors. Importantly, the substantial increase in pairwise agreement when ratings were collapsed into broad agreement and disagreement categories indicates that some assessor variation reflected differences in rating strength within the same overall direction rather than opposing assessments. While such variability is consistent with values reported in comparable evaluations [87,91], it highlights the importance of assessor calibration and clear scoring guidance to minimize avoidable variability in rubric interpretation while preserving legitimate differences in clinical judgment. OMNI is intended to structure and document expert evaluation rather than determine clinical actionability through a predefined assessor agreement threshold. Future implementation studies should establish appropriate numbers of outputs and assessors according to the intended purpose of the evaluation, recognizing that requirements may differ depending on whether the primary objective is to assess interrater reliability, characterize model performance, or compare performance between models. For comparative or clinically consequential applications, OMNI would likely be used by multiple independent, appropriately qualified assessors who have reviewed the instructional materials and completed pre-evaluation calibration, with independent ratings retained or followed by consensus or adjudication where appropriate to the evaluation purpose. However, we also recognize that single-assessor use may still have value for structured reflection, education, or formative assessment, where the aim is to support a standardized consideration of AI output quality rather than generate comparative conclusions.

It is acknowledged that EC contributed across multiple study stages, including rubric development and refinement, pilot assessment, post-pilot feedback, qualitative coding, and statistical analysis, while AS participated in both expert refinement and pilot assessment. Their inclusion as pilot assessors was intentional and enabled evaluation of whether OMNI could be meaningfully applied in practice by those involved in its development. Although prior familiarity had the potential to influence their ratings or feedback, no meaningful differences were identified compared with those of other assessors, as detailed in Tables S4 and S10 in Multimedia Appendix 1. Several additional limitations relate to the scope and design of the pilot. Stage 1 involved a targeted literature synthesis rather than a formal systematic or scoping review; consequently, relevant literature may have been missed. The stage 2 process concluded when no further objections were raised. Although the panel remained actively engaged throughout, this approach may conflate silence or deference with genuine agreement. The evaluation focused on publicly accessible foundational models and single-turn query-response interactions, which may not fully capture the iterative or conversational dynamics of real-world AI use [92]. Only one output was generated per prompt, meaning that model variability across repeated generations could not be assessed. Assessment duration and workload were not prospectively measured; future implementation studies should prospectively measure completion time across different health care contexts, recognizing that assessment burden may vary with scenario complexity and output length. Furthermore, while the expert panel included international insight, all assessors in the pilot evaluations were Australian-trained clinicians, and evaluations relied on subjective expert judgment, which may introduce biases related to regional practice norms or professional training. Finally, while OMNI captures observable output behaviors, it cannot directly assess internal model reasoning processes or latent safety mechanisms that are not reflected in the generated response.

OMNI was evaluated in 4 diverse health care scenarios, selected to reflect current and emerging applications of generative AI in health care [3,90,101]. These scenarios align with several task categories within the MedHELM framework, which classifies medical AI applications across clinical decision support, clinical note generation, patient communication and education, medical research assistance, and administration and workflow [3]. However, the scope of generative AI use in health care continues to expand, and future work should evaluate OMNI across a broader range of clinical contexts, care settings, assessor populations, and AI systems to further assess its generalizability and usability. In particular, this study focused on publicly accessible foundational models, and the rubric’s applicability to purpose-built health AI tools, specialized domain-specific models, and more agentic systems remains to be established. These systems may operate under different constraints, workflows, guardrails, and deployment environments [104], and their evaluation may require further refinement of both OMNI and its domains. Relatedly, while OMNI was designed to support structured expert evaluation, an important future application may be at the organizational level, where institutions assess AI systems for procurement, governance, deployment, and implementation [7,105]. In this setting, OMNI may help support clinician and other domain expert input alongside multidisciplinary administrative, IT, and oversight teams evaluating AI systems against context-specific requirements [7,105]. Structured expert-led evaluation frameworks such as OMNI may also provide a foundation for hybrid approaches that combine human assessment with automated methods, such as LLM-as-a-Judge systems, enabling larger-scale evaluation while maintaining clinically meaningful assessment criteria [17].

Further methodological development may also refine OMNI itself. While the 10 domains included in the current framework represent core constructs for evaluating generative AI outputs, additional “plug-in” domains were identified during the literature synthesis, expert consensus process, and assessor feedback that may be relevant depending on the evaluation setting. These include dimensions such as consistency and robustness [8,14], equity-related harms, fairness or biases [93-96], deployment metrics (such as cost and latency) [8], regulatory compliance [96], trust [14,94], or user satisfaction [14,96]. A comprehensive list of these additional domains for consideration is provided in Table S11 in Multimedia Appendix 1. More broadly, future work may need to regularly review and refine OMNI’s domains and the guiding examples used to support scoring within each domain as new health care applications, models, and implementation contexts emerge.

In conclusion, this study developed OMNI, a multistage, designed rubric that provides health care professionals, educators, administrators, and AI experts with a standardized, accessible, and practical tool for expert-led evaluation of generative AI outputs in health contexts. It notably reduces barriers to meaningful assessment by guiding evaluation across 10 important domains and supporting structured feedback across emerging applications of generative AI in clinical decision support, clinical note generation, patient and clinician communication and education, medical research, and administration. The pilot application demonstrated that OMNI could identify model strengths and significant limitations across evaluation domains in individual scenarios, generating the detailed, expert-authored feedback needed to meaningfully improve these systems. As generative AI becomes increasingly pervasive across health contexts, OMNI offers a standardized yet adaptable foundation that can be extended to new scenarios and settings. Furthermore, OMNI supports expert involvement, facilitating safe deployment and helping to optimize the benefits of these promising technologies. Further work is needed to assess its applicability to purpose-built health AI tools, agentic systems, and organizational evaluation settings.